Evaluating success in autonomous multi-robot teams: experiences from ALLIANCE architecture implementations

Quantitatively evaluating the e ectiveness of software architectures for multi-robot control is a challenging task. Exacerbating the problem is the fact that architectures are typically constructed to address di erent design goals and application domains. In the absence of benchmarks that capture the variety of issues that arise in multi-robot coordination and cooperation, the system developer can only evaluate an architecture for its own qualities. In this short article, we summarize the metrics of evaluation that we utilized in applying our ALLIANCE architecture [2] to eight di erent application domains for multi-robot team control. We explore the implications of the metrics we have chosen and o er suggestions on future productive lines of research into metrics for multi-robot control architectures.